Although significant advances in graphical user interfaces have made software applications more accessible and productive for personal computer users, the increased ease of use has fostered a demand to include more and more sophisticated features. When first exposed to a complicated software application, a new user can be confused by the overwhelming number and complexity of features accessible through the menus and toolbars of the user interface. It is not uncommon for a user unfamiliar with the software features to resort to menu surfing, rapidly switching from menu item to menu item, in the hope of discovering how to effect the desired feature. Even when users know one or more ways to perform a task in a software application, they may not realize an efficient approach. Also, user's may know how to perform tasks, yet instead of performing the task themselves, they would rather have a system proactively determine their needs and perform or offer to perform operations, such as launching applications that will soon be needed, exiting applications, and prefetching files or information from distant servers to make them available more efficiently when they are requested.
Determining in an automated manner the best actions to perform or the best information to provide users while they work with software requires the development of computational methods that operate in conjunction with software programs and that have the ability to identify a user's needs, intentions, or goals from aspects of a user's background and actions. Such methods may be derived from the construction and use of models that explicitly links the users' needs to their backgrounds and actions.
There have been several studies of the use of models for reasoning about the intentions of people performing a variety of tasks, and for making available advice or assistance based on these models. Researchers studying the use of computers in education have attempted to construct programs that use models of users to determine the source of misunderstanding and the best way to tutor a student. These models can look at the answers users give to questions and other challenges. The Strategic Computing Initiative of the 1980s focused in part on methods for doing reasoning about a pilot's intentions from a diverse set of inputs, and for providing them with the best assistance with piloting and warfighting with fighter jets. Within the many such "pilot associates" projects, engineers attempted to design systems that could reason about the intentions of users. Typically, rule-based or logical approaches were used for determining or prioritizing options.
There has also been some work in the use of probabilistic models in the modeling of user's needs for such tasks as monitoring complex, time-critical applications. Some of the earliest discussion of probabilistic dependency models, with applications to enhancing the display of information, severe in projects experimenting with the modeling of users for controlling the information displayed to aircraft pilots. Probabilistic inference was explored as a means for controlling the information being displayed to pilots. Other work included using decision-theoretic methods to reason about the most important information to display to flight engineers at NASA Mission Control Center. The methods consider context, user expertise level, and telemetry from the Space Shuttle to make these decisions, and take advantage of time-criticality and probabilistic models of user beliefs and intentions. Other related work has explored the best kind of information and explanations to display to pilots given various tradeoffs in display. In other work, researchers have explored the use of probabilistic models in assisting computer users working with command line input systems of a computer operating system.
There have been several projects focusing on the use of sets of logical rules and heuristics based on logical rules for providing users with assistance based on the context and activity. For instance, the common context-sensitive help that is made available in applications written for the Microsoft Windows operating system is accessed by selecting a graphical object and clicking a standard help command (e.g., a function key on the keyboard). Help information is then provided which describes the functionality of the graphical object such a toolbar. Another example is the TipWizard system in Microsoft Excel products which employs a large set of logical rules that continue to scan for sequences of user events and provide assistance to users on performing tasks more efficiently. In work at Apple Computer described by U.S. Pat. No. 5,390,281 to Luciw et al., sets of rules are used for providing assistance to users on tasks by employing the heuristic of comparing the number of conditions for a rule to fire with the specific number of conditions that are seen in a particular situation and for interpreting this fraction as a strength associated with the action indicated by the rule.
There has also been work on more sophisticated probabilistic models that consider the uncertainty in user goals and needs for software assistance. The Answer Wizard feature of Microsoft Office products on the use of probabilistic models for interpreting the words in a user's free-text query with a focus on assisting users with using computer software applications. In this work, the uncertain relationship between sets of related words that might be seen in a query and various kinds of assistance are modeled with probability. This work has been described in U.S. patent application Ser. No. 08 400,797.
However, none of these former efforts have employed sophisticated methods for reasoning under uncertainty about a user's needs for assistance by considering multiple pieces of evidence, including information about a user's background and a user's activities that are autonomously sensed by the program, and, then, have offered to provide relevant assistance to a user based on this reasoning. Nor have these systems considered monitoring user interaction with a software program so that, when a user does inquire about help directly with free-text queries, the system combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context.
What is desirable is an intelligent user assistance facility that autonomously senses that the user may need assistance in using a particular feature or to accomplish a specific task, and that offers to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities. What is also desirable is an intelligent user assistance facility that combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. This invention solves the problem of autonomously sensing that a user may need assistance in using a particular feature or to accomplish a specific task, and offering to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities. This invention also solves the problem of combining the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. In addition, the invention is able to monitor and perform inference about several classes of events, including the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information.